Skip to content

feat(autobahn): prepare a restored node's epoch registry from a checkpoint - #4860

Open
blindchaser wants to merge 15 commits into
mainfrom
yiren/giga-ss5-epoch-seed
Open

blindchaser wants to merge 15 commits into
mainfrom
yiren/giga-ss5-epoch-seed

Conversation

@blindchaser

@blindchaser blindchaser commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A node restored from a FlatKV snapshot at height H must know the committees of the epochs around H before it can verify the next blocks. This PR builds that epoch registry from the epoch_registry row in the snapshot. It also makes AutobahnLayout list every store dir that node startup opens, so seid statesync import can require all of them to be empty.

How it works:

  • epoch.NewRegistryFromSnapshot opens a registry from the row value. It runs the same checks that NewRegistry runs on a loaded snapshot, and with a state dir it persists the row there. The decode and the file format stay in the epoch package.
  • node.OpenCheckpointEpochs reads the row from the FlatKV snapshot dir and opens the registry in memory. It must hold the epoch of H and the next epoch, and an H on the last road of an epoch is refused.
  • node.InstallEpochRegistry removes <psd>/epochs and writes the row there with the same constructor. It can run again with the same row, so a rerun after a crash is safe.
  • With no row, the registry holds only the genesis epochs 0 and 1. Consensus does not write the row yet, so today a restore works only at an H in epoch 0.
  • Node startup and the layout build the storage config through one function, autobahnStorageConfig, so the layout lists the dirs that startup really opens.

For reviewers

  • Consensus: please check NewRegistryFromSnapshot in epoch/registry.go. Is the name good? Should the layout keep epoch.SnapshotDir for the <psd>/epochs path?
  • Consensus: please confirm that the startup refactor in node/setup.go and p2p/giga_router_common.go does not change behavior.
  • Consensus: please confirm the epoch rule (the epoch of H and the next epoch, and H does not close an epoch), and that the layout covers every store of a fullnode.
  • Consensus: the genesis check binds the row to this chain only while it holds epoch 0 or 1. We rely on AppQC(H+1) to catch a wrong state. Please confirm that this is enough.

Test plan

  • Layout test: opens the storage manager as startup does, and fails on any dir under the persistent state dir that the layout does not cover.
  • Constructor tests: the source epochs in memory, a persisted row that NewRegistry reads back over an older snapshot, the genesis of another chain, a row pruned past genesis, and a malformed row that writes nothing.
  • Registry tests: no row, an empty or malformed row, the last road of an epoch, a missing next epoch, a pruned epoch, a row from another chain, and a second install of the same row.
  • A data-state test starts a restored node on the epoch of the restored road.
  • scripts/ramtest.sh -race on node, autobahn/epoch, autobahn/data, evmonlyapp and p2p; make fmtcheck and golangci-lint v2.14.0 are clean.

blindchaser and others added 6 commits November 13, 2026 17:39
KVImporter now holds at most 256 MiB of key and value bytes that are not yet flushed. When the
budget is full, AddNode waits for a flush to return bytes. Each worker also flushes when it holds
budget/16 bytes, so the four workers together hold less than a quarter of the budget, and a node
larger than half the budget is refused. A waiting AddNode therefore always waits on bytes that are
in a channel or in a flush, and those bytes always come back, so the import cannot deadlock.

The worker threshold is 16 MiB, a quarter of the default 64 MiB memtable. Pebble keeps a batch of
half a memtable or more on the Go heap until it is written to disk, after the commit that returns
its bytes to the budget. In an import of 512 MiB of 24 KiB rows, the peak live heap was 0.7 to
1.0 GiB with a 32 MiB threshold and 0.4 GiB with 16 MiB.

An error broadcasts under the budget lock, so a waiting AddNode cannot miss it. A release that takes
the count below zero fails the import instead of being hidden.

Co-authored-by: Cursor <cursoragent@cursor.com>
ExportSnapshotDir streams the live rows of a closed snapshot dir in physical-key order, merging the
four data databases. SnapshotDirStats returns what the snapshot metadata records: the row count, the
key and value bytes, and the state hash, which is the checksum of the store-wide LtHash. The export
counts the rows and bytes it emits, and fails when they differ from the metadata.

ReadSnapshotMiscValue reads one non-EVM row. It fails when the dir has no misc database, so a wrong
path is not read as a snapshot without the row.

The state-sync snapshot tool uses these functions on a staged checkpoint where no CommitStore is
open.

Co-authored-by: Cursor <cursoragent@cursor.com>
KVImporter now holds at most 256 MiB of key and value bytes that are not yet flushed. When the
budget is full, AddNode waits for a flush to return bytes. Each worker also flushes when it holds
budget/16 bytes, so the four workers together hold less than a quarter of the budget, and a node
larger than half the budget is refused. A waiting AddNode therefore always waits on bytes that are
in a channel or in a flush, and those bytes always come back, so the import cannot deadlock.

The worker threshold is 16 MiB, a quarter of the default 64 MiB memtable. Pebble keeps a batch of
half a memtable or more on the Go heap until it is written to disk, after the commit that returns
its bytes to the budget. In an import of 512 MiB of 24 KiB rows, the peak live heap was 0.7 to
1.0 GiB with a 32 MiB threshold and 0.4 GiB with 16 MiB.

An error broadcasts under the budget lock, so a waiting AddNode cannot miss it. A release that takes
the count below zero fails the import instead of being hidden.

Co-authored-by: Cursor <cursoragent@cursor.com>
ExportSnapshotDir streams the live rows of a closed snapshot dir in physical-key order, merging the
four data databases. SnapshotDirStats returns what the snapshot metadata records: the row count, the
key and value bytes, and the state hash, which is the checksum of the store-wide LtHash. The export
counts the rows and bytes it emits, and fails when they differ from the metadata.

ReadSnapshotMiscValue reads one non-EVM row. It fails when the dir has no misc database, so a wrong
path is not read as a snapshot without the row.

The state-sync snapshot tool uses these functions on a staged checkpoint where no CommitStore is
open.

Co-authored-by: Cursor <cursoragent@cursor.com>
…to cover them

The layout now lists the block DB, receipt store, state store and app-hash vault dirs beside
FlatKV, so state sync can require them empty on restore. It takes the storage config and the
genesis doc, because the storage config requires the initial height. The new test opens the
storage manager the way node startup does, and fails on any entry under the persistent state
dir that the layout does not cover.

Co-authored-by: Cursor <cursoragent@cursor.com>
PrepareEpochRegistry reads the epoch_registry row from a staged FlatKV checkpoint, the
PersistedEpochRegistry that consensus commits in EpochState, and writes it as an epoch registry
snapshot in a staging dir. The registry must open from genesis the same way node startup opens it,
through p2p.OpenEpochRegistry, which BuildDataState now also uses. It must hold the epoch of the
checkpoint height and the epoch after it, and a height at the last road of an epoch is refused.
With no row the registry holds the genesis epochs, and an empty row is an error. The genesis check
binds a row to this chain only while the row holds epoch 0 or 1.

InstallEpochRegistry moves the staged snapshot to the layout's EpochRegistryDir. The layout also
names the state sync proof file that a restore installs.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedNov 17, 2026, 4:41 AM

@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.25478% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.64%. Comparing base (552d731) to head (06bbf5a).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
sei-db/state_db/sc/flatkv/snapshot_read.go 80.57% 27 Missing ⚠️
sei-tendermint/internal/p2p/giga_router_common.go 47.05% 18 Missing ⚠️
sei-tendermint/node/epoch_seed.go 83.87% 5 Missing ⚠️
sei-tendermint/node/testonly.go 0.00% 4 Missing ⚠️
sei-tendermint/node/setup.go 62.50% 3 Missing ⚠️
sei-db/state_db/sc/flatkv/importer.go 95.91% 2 Missing ⚠️
sei-tendermint/internal/autobahn/epoch/registry.go 93.33% 2 Missing ⚠️
sei-tendermint/internal/evmonlyapp/epoch_state.go 83.33% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4860      +/-   ##
==========================================
+ Coverage   61.59%   61.64%   +0.04%     
==========================================
  Files        2079     2082       +3     
  Lines      161891   162138     +247     
==========================================
+ Hits        99715    99946     +231     
- Misses      62171    62187      +16     
  Partials        5        5              
Flag Coverage Δ
sei-chain 59.84% <73.60%> (+0.02%) ⬆️
sei-db 75.17% <ø> (ø)
sei-db-state-db 81.62% <84.65%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/state_db/sc/flatkv/store_meta.go 78.23% <100.00%> (-0.38%) ⬇️
sei-tendermint/internal/autobahn/epoch/snapshot.go 97.36% <100.00%> (+0.03%) ⬆️
sei-tendermint/internal/autobahn/epoch/testonly.go 78.26% <100.00%> (+6.83%) ⬆️
sei-tendermint/node/autobahn_layout.go 94.44% <100.00%> (+2.77%) ⬆️
sei-tendermint/internal/evmonlyapp/epoch_state.go 92.30% <83.33%> (-7.70%) ⬇️
sei-db/state_db/sc/flatkv/importer.go 94.67% <95.91%> (+0.25%) ⬆️
sei-tendermint/internal/autobahn/epoch/registry.go 96.55% <93.33%> (-1.98%) ⬇️
sei-tendermint/node/setup.go 64.74% <62.50%> (+0.09%) ⬆️
sei-tendermint/node/testonly.go 0.00% <0.00%> (ø)
sei-tendermint/node/epoch_seed.go 83.87% <83.87%> (ø)
... and 2 more

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blindchaser and others added 5 commits November 16, 2026 15:14
…seed

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	sei-tendermint/internal/evmonlyapp/app_test.go
readCommittedVersion and the snapshot dir readers held the same
read-only pebble.Open. Both now call openPebbleReadOnly, which returns
the Pebble error unchanged, so the error text and the
ErrDBDoesNotExist check in readCommittedVersion are the same as before.

Co-authored-by: Cursor <cursoragent@cursor.com>
A flush returns its rows' bytes to the import byte budget, but resetting
ltMutations to length zero kept the rows reachable through the backing
array. A byte-triggered flush can be shorter than the batch before it, so
the next batch did not overwrite every slot, and rows that the budget no
longer counted stayed in memory.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	sei-db/state_db/sc/flatkv/import_export_test.go
#	sei-db/state_db/sc/flatkv/snapshot_read.go
@blindchaser
blindchaser changed the base branch from main to yiren/giga-ss5-flatkv November 16, 2026 21:09
State the crash contract of InstallEpochRegistry, return the genesis
validators in key order, and rename the test for a row pruned past
genesis so that its name states the limit.

Co-authored-by: Cursor <cursoragent@cursor.com>
@blindchaser
blindchaser marked this pull request as ready for review November 16, 2026 23:28
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T23:31:24.813784Z 988890c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cursor

cursor Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Summary

High Risk
Changes consensus-critical epoch restoration, checkpoint trust boundaries, and FlatKV import memory behavior; incorrect validation or layout gaps could break state sync or allow wrong committee sets until later AppQC checks.

Overview
Adds state-sync preparation so a node restored at height H can rebuild the epoch registry from the FlatKV checkpoint’s autobahn/epoch_registry row: OpenCheckpointEpochs validates the row (decode, genesis binding for epochs 0–1, epoch at H plus the next epoch, reject last road of an epoch / empty row), and InstallEpochRegistry persists it under the persistent state dir. RestoreEpochRegistry / NewRegistryFromSnapshot share the same validation path as normal startup, and ReadCheckpointEpochRegistryRow reads the row from a linked snapshot dir.

AutobahnLayout now lists all store paths node startup opens (FlatKV, block DBs, receipts, state store, app hash vault, epoch snapshots, restore/proof files), built via shared autobahnStorageConfig so layout and startup stay aligned—supporting import rules that require empty dirs.

On the FlatKV side, new ExportSnapshotDir / SnapshotDirStats / ReadSnapshotMiscValue validate exported rows against snapshot metadata and enable checkpoint introspection. The KV importer gains a 256 MiB byte budget with worker flushes on bytes (not only pair count), backpressure, and per-node size limits.

Extensive tests cover layout coverage, registry edge cases, restored-node anchoring, and importer/snapshot parity.

Reviewed by Cursor Bugbot for commit 06bbf5a. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 988890c0d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sei-tendermint/node/epoch_seed.go Resolved
seidroid[bot]
seidroid Bot previously requested changes Nov 16, 2026

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builds a restored node's epoch registry from the FlatKV checkpoint's epoch_registry row, and makes AutobahnLayout list every store dir that startup opens by building the storage config through the same function startup uses. The crash-retry path of InstallEpochRegistry, which its own doc tells callers to use, can wipe the installed registry and still report success, so this is request_changes. That finding came from codex and I confirmed it against the tree; it was the only reading, and the startup refactor looks behavior-preserving. There is no Go toolchain in this sandbox, so the tests were not run.

Blocking

1 finding on the changed lines, as inline comments.

seidroid review · decision request_changes · session 1ae92a41958849c4af27d417a5ce2086 · turn resp_claude_b4087d7fe11f925b667305594404e0a7 · item 93ee4fbb60d75f138e58637962544178

Findings: 1 blocking | 0 non-blocking | 1 posted inline

Comment thread sei-tendermint/node/epoch_seed.go Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 988890c. Configure here.

Comment thread sei-tendermint/node/epoch_seed.go Resolved
InstallEpochRegistry removed the installed snapshot before it checked
the staged one, and ignored a missing source, so a second call with the
same staging dir left only the genesis epochs and reported success. It
now refuses a missing staged snapshot before it removes anything.

Co-authored-by: Cursor <cursoragent@cursor.com>

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InstallEpochRegistry now checks that the staging dir holds a prepared snapshot before it removes the installed one, and it no longer swallows ErrNotExist. A new test shows a second install from the same staging fails and keeps the installed registry, so my blocker is fixed and nothing else blocks. Nothing else in the diff changed since my last review; codex found nothing in this round, which matches my reading, and there is still no Go toolchain here, so the tests were not run.

seidroid review · decision approve · session 1ae92a41958849c4af27d417a5ce2086 · turn resp_claude_8354077b2317694be5ea1786aa0eda4d · item 62ee35e828c25299a930ac6fb0647504

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@seidroid
seidroid Bot dismissed their stale review November 16, 2026 23:55

Superseded: the latest review found nothing blocking in this change.

…omments

Co-authored-by: Cursor <cursoragent@cursor.com>
)

// SnapshotDir returns the dir under stateDir that holds the epoch registry snapshot.
func SnapshotDir(stateDir string) string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we reading a file from directory? I suppose writing and reading of epoch snapshots should all go through storage manager?

The previous WAL files are temporary measure, they should go away very soon, restored by the values in FlatKV.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Base automatically changed from yiren/giga-ss5-flatkv to main November 17, 2026 00:56
…alue

Add epoch.NewRegistryFromSnapshot, which opens a registry from the
FlatKV epoch_registry value with the same checks as NewRegistry and
persists it when a state dir is set. Remove epoch.RestoreSnapshot.

node.OpenCheckpointEpochs opens the checkpoint registry in memory, and
InstallEpochRegistry writes it into the persistent state dir with the
new constructor. There is no staging dir and no rename, and an install
can run again with the same CheckpointEpochs.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants